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Abstract 

We consider zero-sum repeated games with incomplete information on both sides, where the 
states privately observed by each player follow independent Markov chains. It generalizes the 
model, introduced by Aumann and Maschler in the sixties and solved by Mertens and Zamir 
in the seventies, where the private states of the players were fixed. It also includes the model 
introduced in Renault [20], of Markov chain repeated games with lack of information on one 
side, where only one player privately observes the sequence of states. We prove here that the 
limit value exists, and we obtain a characterization via the Mertens-Zamir system, where the 
"non revealing value function" plugged in the system is now defined as the limit value of an 
auxiliary "non revealing" dynamic game. This non revealing game is defined by restricting 
the players not to reveal any information on the limit behavior of their own Markov chain, as 
in Renault 2006. There are two key technical difficulties in the proof: 1) proving regularity, in 
the sense of equicontinuity, of the T-stage non revealing value functions, and 2) constructing 
strategies by blocks in order to link the values of the non revealing games with the original 
values. 

Key words: repeated games; incomplete information; zero-sum games; Markov chain; lack of 
information on both sides; stochastic games; Mertens-Zamir system. 

1 Introduction 

The pioneering work of Aumann and Maschler in the 1960s introducing Repeated games with 
incomplete information (see [1] for a re-edition of their work) was widely studied and extended. 
One of their most famous results concerns the model of two-player zero-sum repeated game 
with lack of information on one side and perfect observation: At the beginning of the game, a 
state variable k is chosen at random in a finite set K using some probability p, and announced 
to Player 1 only. A finite zero-sum game Gk depending on the state variable is then played 
repeatedly and after each stage the actions played are observed by both players. The famous 
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cavu theorem of Aumann and Maschler then reads as fohows: the value of the infinitely 
repeated game exists and is characterized as the concave hull of the value of an auxiliary game 
called the non-revealing game. In a previous work of Renault [20], this model was extended 
in the following direction: instead of being fixed once for all at the beginning of the game, 
the state variable was assumed to evolve according to a Markov chain of initial probability 
p and transition matrix M over K. Renault proved that in this model, called Markov chain 
game with incomplete information on one side, the value of the infinitely repeated game exists 
and gave a characterization for it, very close to the one of Aumann and Maschler. The main 
difference was in the definition of the non-revealing game: it was defined in Renault [20] 
as a dynamic game with infinite length where Player 1 was constrained not to reveal any 
information on the asymptotic behavior of the Markov chain rather than on the current state 
variable itself. The model of Markov chain game with incomplete information on one side was 
also studied by Neyman [16] who proved the existence of an optimal strategy for Player 1 and 
also proved a generalization of the existence of the value in case of imperfect observation of 
actions. 

A more involved model also introduced in [1] is the model of two-player zero-sum repeated 
game with lack of information on both sides and perfect observation: At the beginning of the 
game, two states of nature k and I are chosen independently in some finite sets K and L, 
according to some probabilities p and q. The state k is announced to Player 1 only, and the 
state I is announced to Player 2 only. These states of nature determine a finite zero-sum game 
Gk,i which is then played repeatedly; after each stage the actions played are observed by both 
players. Aumann and Maschler showed that the infinitely repeated game may have no value. 
Mertens and Zamir [14], showed the existence of the limit value (i.e. the existence of a limit 
for the sequence of values of finitely repeated games when the number of repetitions goes to 
infinity) in this model and gave a characterization for it (see the system (1) later) based on a 
system of functional equations acting on the value of the non-revealing game. 

The results of Aumann and Maschler and of Mertens and Zamir led to a great number 
of works dedicated to generalizations of this model, or close extensions of it. Let us cite for 
games with incomplete information on one side, Kohlberg [10] for an explicit construction of 
an optimal strategy for Player 2, De Meyer and Rosenberg [4] and Laraki [12] for alternative 
proofs of existence of the limit value based on duality, Gensbittel [5] for the extension to 
infinite action spaces, or Sorin [26], Hart [7], Simon et al. [2.3], Renault [18] [19] for the 
nonzero-sum case. 

Concerning incomplete information on both sides, let us mention the proof of existence of 
asymptotically optimal strategies by Heuer [8], the extension of the Mertens-Zamir system by 
Sorin [27], the study of an abstract game called the splitting game related to the system of 
functional equations by Laraki [13], alternative proofs of existence of the limit value based on 
the so-called operator approach by Rosenberg and Sorin [22], or more recently on continuous- 
time approach by Cardaliaguet, Laraki and Sorin [3]. 

We only consider here two-player zero-sum games and generalize the model of repeated 
game with lack of information on both sides to the case where the states of nature {k, I) are no 
longer fixed at the beginning of the game, but evolve according to given independent Markov 
chains {kt)t>i and {lt)t>i- At the beginning of each stage, kt is observed by Player 1 only, 
and It is observed by Player 2 only. We call such games Markov chain games with lack of 
information on both sides. Note that this model admits as a special case the model of repeated 
game with incomplete information on both sides, in which the value of the infinitely repeated 
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game may not exist. In this paper, we generalize both the proofs of Renault [20] and Mertens 
and Zamir [14] and show the existence of the limit value for Markov chain games with lack 
of information on both sides. We also give a characterization for this limit value based on a 
system of functional equations, similar to the one introduced by Mertens and Zamir, and on 
the generalized notion of non-revealing games as introduced by Renault. 

However, as already mentioned in these two works, our expression of the value cannot 
be easily computed from the basic data of the game. It was already noticed in [14], where 
several examples of computations are given, underlying that the limit value may be outside 
of the class of semi-algebraic functions, which contrasts with properties of the limit value for 
stochastic games (with complete information and finite state and action spaces). Moreover, 
the problem of computation of the value of non-revealing games for Markov chain games 
with incomplete information on one side was already mentioned by Renault who gave an 
example which appeared to be very difficult to compute, except for very particular values of 
the transition matrix as shown in the work of Horner et al. [9] . 

Section 2 of this paper contains the model. Section 3 contains preliminary results and 
notations. A few important examples, and the main ideas of the proof, are presented in 
section 4. In section 5, we define and study a notion of nonrevealing strategies via projection 
matrices that will be used to quantify a notion of relevant information for both players. We 
introduce the auxiliary games called nonrevealing games, where both players are restricted 
to play a nonrevealing strategy, and prove that they have a limit value. In section 6, we 
introduce a system of functional equations very similar to the system introduced by Mertens 
and Zamir and prove that the limit value is the unique solution of this system, which is itself 
parameterized by the limit value of the nonrevealing game. Section 7 contains the proof of 
some technical results. We conclude in section 8 with a list of open questions related to the 
possible extensions of the model. 

2 Model 

Given a finite set S, A{S) denotes the set of probabilities over S and the cardinal of S is, 
with a slight abuse of notations, also denoted by S. The set of positive integers is denoted 
IN*. 

We consider a zero-sum game between 2 players with action sets / and J, sets of states 
K and L, where /, J, K, L are disjoint finite non empty sets, and payoff function 

g:K X Lx I X J — y [-1,1]. 

M and N are given Markov matrices on K and L respectively, i.e. M = {Mi^^y)k^}^iizK is a 
K X K matrix with non-negative entries and such that for all k £ K, Ylk'eK ^k,k' = 1> and 
similarly for A'^. The letters p £ A (AT) and q £ A(L) denote initial probabilities. 

The game is played by stages in discrete time. We are given two independent Markov 
chains: {kt)t>i with initial distribution p and transition matrix M, and {lt)t>i with initial 
distribution q and transition matrix N. At the beginning of every stage t > 1, Player 1 
observes kt and Player 2 observes It- Then both players simultaneously select an action in 
their action set, if {it,jt) in / x J is played then Player I's payoff for stage t is g{ktJt,H, jt)- 
Then {it,jt) is publicly observed and the play goes to stage t + 1. Notice that the payoff 
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g{h,lt,H,jt) is not directly observed and may not be deduced by the players at the end of 
stage t. 

We denote by S and T the sets of behavior strategies of the players. Formally, for t > 1, 
let Ht = {I X jy denote the set of histories of actions of length t, with the convention 
^0 = {0}- A strategy o" € S is a sequence (o"f)f>i, where at is a map from Hf-i x to 
A(/). For T G IN*, the set Et denotes the set of T-stage strategies for Player 1, that is of 
finite sequences (ai, ..,(7t) induced by elements of S. T and 7t are defined similarly. 

The value of the T-stage game is denoted vt{p, q), i.e. 

vt{p-,q) = maxmin7j,''^(c7, r) = minmax7j,'''(c7, r), 

where ^^{(j,t) = lEpg,jrYYlt=i9{^t,h,'i't,jt), the expectation being taken with respect to 
the probability IPp,q,a,T induced by {p,q,a,T) on the set of plays Q = (K x L x I x J)^. 
Elements of A{K) are seen as row vectors, so that if p' in A{K) is the law of the state kx of 
stage T, then p'M is the law of the following state fcr+i (similarly elements of A.{L) are seen 
as row vectors). We have the standard recursive formula, for all T > 0: 

{T +l)vT+i{p,q) = max mm I G{p,q,x,y) + T V x{p){i)y{q){j)vT{p{x,i)M,q{y,j)N) 

= mm max I G{p,q,x,y) + T V x(j)){i)y{q){j)vT{p{x,i)M,q{y,j)N) 

where G{p,q,x,y) = Y^k,l,i,jP^1^^'^i^)y\j)9{k,l,i,j), 

Vi G I, = and = (^^y||) ^ A(i^) if x(p)(i) > 0, 

Vi G J, = Y,q'y\j), and ^(y,;) = (^||^)^ ^ ^ A(^) if > 0, 

and p{x,i) (resp. q{y,j)) is defined arbitrarily if = (resp. y{q){j) = 0). If at some 

stage t the belief of Player 2 on the current state kf in K is represented by p, and if Player 1 
plays the mixed action if his current state is k, then x{p){i) will be the probability that a 
given action i in / is played at this stage. After observing i, p{x, i) represents the new belief 
of Player 2 on kt, and p{x, i)M is the belief of Player 2 on fct+i. 

Theorem 2.1. v{p,q) =limvT{p,q) exists. 

Moreover, we will provide a characterization for the limit function v. 

Definition 2.2. Let f be a function from A{K) x A(L) to M. 

• f is I-concave if for all q G A(L) the function p i— t- f{p, q) is concave on A{K), 

• f is Il-convex if for all p G A{K), the function q i-)- f{p, q) is convex on A(L), 

• f is balanced if for all {p,q) G A{K) x A(L).- 

^(P) q) = w{pM, qN). 
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Notation 2.3. We denote by W the set of continuous functions w : A(K) x A(L) — [—1, 1]. 

We will soon assume (w.l.o.g., see sections 3 and 4) that both Markov chains on K and L 
are recurrent and aperiodic. In this case we will show in Theorem 6.1 that v can be expressed 
as: 



inf 



sup 



w £ W, w is /-concave, balanced and 
V(p,g) G A{K) X A(L), w{p,q) > veyinMax{w,v){p,q] 

w £ W, w is //-convex, balanced and 
y{p,q) € A(/C) X A(L), w{p,q) < cavjMm{w,v){p,q) 



where v is the "nonrevaling limit value" that will be defined in Proposition 5.17 and where 
cavj and vex// denote respectively the concave and convex hull with respect to the first and 
second variable (see Notation 3.3 below). 



3 Preliminaries 

In all the paper the sets A{K), A(L) and A{K) x A(L) are endowed with the L^-norm, 

\\p-p'\\ = Ek Ip^-p'^l, \\{p,q) - {p',q')\\ = Ek \p'-p"\ + El W - /I etc... 

Lemma 3.1. For each T , vt is I-concave, Il-convex and 1-Lipschitz. 

Proof. The game TT{p,q) can be seen as a game with incomplete information on one side 
(the state space being either K or L). The result follows therefore from Propositions 2.1 and 
2.2 in [25]. □ 

Lemma 3.2. For all T > 1, and {p,q) € A{K) x A{L), we have: 

2 2 

\vT+i{p,q) -VT{p,q)\ < ^, \vT+i{p,q) - VT{pM,qN)\ < -, 

and consequently \vT{p,q) — VT{pM,qN)\ < ^. 

Proof. Let o" G S be an optimal strategy in the game Tt{p, q), then for all t G T, 

T 1 2 

which implies VT+i{p,q) > VT{p,q) — The reverse inequality is obtained by inverting the 
role of the players. Using now the recursive formula, let x be defined by x''{i) = j for all 
{k, i) £ K X I. By construction p{x, i) = p for all i G / and we have 

1 T 

T + i ~^ r + 



VT+i{p,q)>-7^r—r + 7fr--r inf V y(g')(j>T(pA/, g(y, j)A^) 
T+1 T + 1 y^^^j^K 



1 T 

> \ VT(pM,qN), 

where the last inequality follows from Jensen's inequality since vt is //-convex and by con- 
struction X^jgj y(9)(j)^(y, j) = q- The reverse inequality is obtained by inverting the role of 
the players. The last statement follows from the triangle inequality. □ 
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If some state k is transient, then for each e there exists To such that ]P(\/T > Tq, ^ 
k) > 1— e. Both players can wait until stage Tq and enter a game where state k has disappeared 
with high probability, so we can assume w.l.o.g. that there is no transient state. From now 
on, we assume that all states in K and L are recurrent. 

We now take care of periodicity. Some states may a priori be periodic, let Tq be a common 
multiple of all periods in the chains induced by M and A'^. Prom the preceding lemma, we have 
\\m.sn\)rpVTTo{p-,<l) = limsupj^ z;t(p, Q'), and lim inf^ ^^^0 (p, g) = liminfr?;r(p, g), so to study 
\\m.TVT we can consider plays by blocks of Tq stages (so that one stage now corresponds to 
To original stages). The Markov chains with transitions matrices M"^" and N"^" are aperiodic, 
and this consideration is without loss of generality. 

Summing up, from now on^ we assume w.l.o.g. that both Markov chains are recurrent 
and aperiodic. We still use the letters M and N to denote the (now recurrent aperiodic) 
transitions matrices. We have the convergence of M* to a stochastic matrix B such that 

= B = BM = MB. The state space K can be partitioned into recurrence classes K{\), 

K(rM), where rM is the number of recurrence classes associated to M. Each class K{r) 
has a unique invariant measure p*{r), and the k-th. row of B corresponds to p*{r) where K{r) 
is the class containing k. Invariant measures for B and M coincide, the set of those measures 
is the convex hull of {p*(l), ...,p*{rM)} denoted by P* . Although B is not invertible if there 
are less recurrence classes than states, for p* in P* we write B^^{p*) = {p £ A{K),pB = p*}. 

Regarding the Markov chain associated to the states in L, we proceed similarly and use 
the following notations. There is convergence of A^* to a stochastic matrix C. The state space 
L is partitioned into recurrence classes T(l), L{rL). Each class L{r) has a unique invariant 
measure q*{r), and the l-th row of C corresponds to q*{r) where T(r) is the class containing 
I. Invariant measures for C and coincide, the set of those measures is the convex hull of 
{q* {!),..., q*{rL)} denoted by Q*. For q* in Q*, we write C-'^{q*) = {q e A{K),qC = q*}. 

We will extensively use the following notations. 

Notation 3.3. If f : X — > M is a bounded function defined on some convex set X , cav(/) 
denotes the smallest concave function greater or equal to f , and vex(/) denotes the largest 
convex function lower or equal to f . If f is defined on a product of convex sets X xY, for all 
{x,y) G X X Y, cav/(/)(x,y) denotes the concave hull of the partial function f{.,y) at point 
x. vex//(/) is defined symmetrically. 

Notation 3.4. For any continuous function f : A{K) x A{L) — > M, we write: 



4 Examples and overview of the proof 

Let us at first emphasize that, in contrast with repeated games with incomplete information, 
if Player 1 does not use his information on the states {kt)t>i, beliefs of Player 2 over states 

^except in the examples D and E of section 4. 




w £ yV, w is I -concave and 
y{p,q) G A{K) X A(L), w{p,q) > YexnMsix{w, f){p, q) 

w £ yV,w is II -convex and ) 
y{p,q) e A{K) X A(T), w{p,q) < caYjMm{wJ){p,q)j 
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evolve across stages according to the transition matrix M. Precisely, under this assumption, 
the belief of Player 2 on the current state kt+i at stage t + 1 (i.e., the conditional law of kt+i 
given the actions played up to stage t) is simply the law of /cj+i, which is pM*. Symmetrically, 
if Player 2 does not use his information on the states {lt)t>ii then the belief of Player 1 on 
at stage t + 1 is qN^. 



Example A: A Markov chain without memory. If both Markov chains are i.i.d. se- 
quences, then the problem reduces to a classical repeated game, by replacing the payoff func- 
tion in pure strategies by its expectation with respect to (kt,lt)- The value of the infinitely 
repeated game exists and equals the value of this auxiliary one-stage game. 

Example B: Irreducible aperiodic Markov chains. Assume that both chains are irre- 
ducible and aperiodic, say 

3 1 



This Markov chain has a unique recurrence class, so rM = = 1 and K = K{1) = L = L{1). 

Unique invariant probabilities are p* = (^,^) and q* = (|,|). and respectively 
converge to: 

S = ( ? ? ) and C 



.5 5/ 

In this case, the limit value exists, and we even have the stronger result that the (uniform) 
value of the infinitely repeated game exists. At first, if Player 1 plays independently of the 
states, the beliefs of Player 2 converge to p* . On the other hand, after any sufficiently long 
number of stages, he can always forget the past actions of Player 2 and act as if his beliefs 
were very close to q*. Choose now some integer Tq such that M^o and iV^° are respectively 
close to B and C, and then Ti much larger than Tq. Let us construct a strategy of Player 
1 as follows: (1) play any strategy independent of the states during Tq stages, then (2) play 
an optimal strategy in the game ^Tiip* ,Q*), and (3) come back to (1). Such a strategy guar- 
antees to Player 1 VTi{p*,q*) up to some arbitrarily small error, hence by choosing Ti large 
enough Player 1 guarantees limsupj^VTip* , q*) the infinitely repeated game. Inverting the 
role of the players implies that the value of the infinitely repeated game exists and equals 
limTVTip* ,q*)- In particular it does not depend on the initial probabilities {p,q). Notice that 
this proof can be generalized to the case of any irreducible aperiodic Markov chain. 

Example C: M and L are the identity matrix. In this case, the problem is a repeated 
game with incomplete information on both sides as studied by Mertens and Zamir [14] since 
the initial state is selected at the beginning of the game and remains constant. In terms of 
Markov chains, the states can be identified with the recurrence classes, and therefore any 
information on the states is an information on the asymptotic behavior of the corresponding 
chain. 

Let us at first define the non-revealing game: it denotes the game where none of the players 
is allowed to use his information on the state variable (i.e. players are restricted tu use strate- 
gies which do not depend on the selected states). For each pair {p,q) of initial probabilities, 
this game can be analyzed as the repetition of the average matrix game "^i^ ip''q''g{k,l, ., .). 
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Consequently, its value is just the value g) = niax^g^(/) minj,gA(j) Sfc ^^^^'^(^i ^ y(j)) 
of the one-stage average game. 

Mertens and Zamir proved that the limit value v is known to exist and is characterized 
as the unique function w £ W solution of the so-called Mertens-Zamir system: 

This system can be recast in a system of functional inequalities: Mertens and Zamir also 
proved that 

V = sup{w\w S W~(n)} = inf{tt;|tt; G VV^(u)}. 

The proof of the second equality was based [14] on game considerations, and as a consequence 
they obtained that supj^lw S W~(/)} = infjtt?!?/; G VV"'"(/)} for any continuous function 
/. This result was later established and extended by the same authors [15], using analytical 
arguments. 

Assuming the equality sup{t(;|t(; G W^(n)} = inf{w|w G W"*"(u)}, the proof of convergence 
of {vt) by Mertens and Zamir can be sketched as follows. 

The main point is to show that for any function w in yV~{u), Player 1 can defend in the 
game of length n with initial probabilities p and q, the quantity w^p, q) up to an error going 
to zero with uniformly with respect to {p, q) and the chosen strategy of Player 2. This can 
be done as follows. The strategy of Player 2 being known, Player 1 can compute after each 
stage t the current pair of beliefs {pt,qt) of both players: pt denotes the conditional law of 
kt+i given past actions and qt denotes the conditional law of k+i given past actions. Then, 
either u{pt,qt) > w{pt,qt) and he plays an optimal strategy in the non-revealing game at 
iPt,Qt), or u{pt,qt) < w{pt.,qt) and he can split his information (think of Player 1 sending a, 
possibly random, message to Player 2) in order to drive Player 2's beliefs at some points pt 
such that u{pt,qt) > w{pt,qt)- The existence of such a splitting is ensured by: 1) the property 
w G W^(n) and 2) the splitting Lemma (see e.g. the Lemmas 6.5 and 6.6 in the present work). 
This strategy would defend w{pt,qt) at stage t + 1 if Player 2 was not using his information on 
the state at stage t + 1. This is almost the case since the error can be bounded by the expected 
-Li-variation of the beliefs \\qt+i — Qt\\- Recall that the sequence of beliefs (qt) is a martingale 
and a classical bound on its Li variation allows to bound uniformly the average error by a 
quantity vanishing with n. Moreover using the properties of w and the construction of the 
strategy of Player 1, the expected value of w{pt,qt) is always greater or equal than w{p,q). 
All this implies that liminf„f„(p, g) > w{p,q), so we obtain liminf„f„ > sup{t(;|t(; G W~(m)}. 
Inverting the role of the players, we also have limsup„v„ < inf{iz;|i/; G W^(m)} which con- 
cludes the proof. 

Example D: A periodic chain Let K be {a, b}, and M be ^ ^ ) 

This Markov chain has a unique recurrence class, which is periodic with period 2. The 
relevant information for Player 2 is not the recurrence class, but the initial state which de- 
termines whether the sequence of states {kt)t>i will be {a, 6, a, 6, ...} or {6, a, 6, a, ...}. By 
considering the auxiliary game in which stages are blocks of length 2 of the initial game, then 
the new transition matrix is M^, hence the identity, and each initial state becomes now a 
recurrence class. The problem can be reduced to that of Example C. As explained in section 
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3, this idea of playing stages by blocks of fixed length can be used in the general case to 
assume without loss of generality that the chain is aperiodic. 



Existence of transient states does not play an important role here. We, however, present 
now an example with a transient state which will be used later (see Remark 6.3). 
Example E: K = {a, 6, c}, and 

M = 




The matrix B = limjM* would in this case be the following: 




The last example below will be used to illustrate several definitions. 
Example F: K = L = {a, b, c}, and 



M = N 




In this case, there are two recurrence classes K{1) = {a, b} and K{2) = {c}. The associated 
invariant measures are p*{l) = (i, i,0) and p*{2) = (0,0, 1), and the limit projection matrix 



IS 



B 



The following figure illustrates some of our notations. 





B-^pB) 



p*(2) 
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Existence of the limit value in case of the last Example F appears to be more difficult, and 
we do not know how to proceed without following the general proof presented in this pa- 
per. Following the general scheme of Renault [20], we consider two kinds of information: 
long-term information (corresponding to the recurrence classes of the two chains, as in the 
Mertens-Zamir case), and short-term information (corresponding to the state variable itself 
within some recurrence class). 

We now come back to the general case, assuming the Markov chains are recurrent and 
aperiodic, and describe the different steps of our proof. 

Overview of the Proof 

By analogy with the proof in Renault [20], we define a non-revealing game, where Players are 
restricted to strategies that do not reveal information on the recurrence classes of the chains. 
Precisely, the strategy of Player 1 has to be such that, after each history of actions of length t, 
the conditional probability that kt+i belongs to any recurrence class K{r) remains unchanged 
(see Definition 5.3). 

This conceptual definition has to be compared with the non-revealing strategies introduced 
by Aumann and Maschler in case each player observes after each stage a signal which is a 
function of the actions played, and which requires that the beliefs of Player 2 on the state k 
remains unchanged. 

The main difference here is that the study of the non-revealing game can not be reduced 
to that of a finite matrix game. Following the scheme of the proof of example B, with 
appropriate modifications, we show that the limit value v of this non-revealing game exists 
and that it depends only on the limit distributions of the two Markov chains {pB, qC). This 
proof requires however a precise analysis of the T-stage non-revealing games, for which we 
prove the existence of the value vt and an appropriate recursive formula. The crucial point 
of the analysis of the non revealing values is to establish that the family {vt)t>i is uniformly 
equi-continuous (this is the aim of the whole section 7), which implies uniform convergence 
of these values to their limit. 

The second main technical difficulty is to link the value of the non revealing games with 
those of the original game, i.e. to adapt the proof of Mertens and Zamir illustrated in example 
C. We use the uniform convergence mentioned above, in particular for sufficiently large Tq, 
vtq is e-close to v. We prove that for any G IV*, and for any balanced function w £ yV~{v), 
Player 1 can defend w{p,q) in Tj^ToiP^Q) up to some error going to zero with n uniformly 
with respect to {p, q) and the chosen strategy of Player 2. The construction of the strategy 
of Player 1 is the same as explained in example C, except that stages are replaced by blocks 
of To stages: Player 1 can compute after each block the current pair of beliefs (PnTo^QnTo) 
of both players and either (1) v{pnTojQnTo) ^ wiPnTo,qnTo) ai^d he plays an optimal strategy 
in the non-revealing game of length Tq at {pnTo,qnTo), or (2) v{pnTo,QnTo) < w{PnTo,qnTo), 
and he can send a random message to Player 2 in order to drive his beliefs at some points 
PnTo such that v{pnToTQnTo) ^ {pnTo t QnTo) and there he plays as in case (1). This strategy 
would defend w^pnTo^QnTo) up to an error of e on the n-th block of size Tq if Player 2 was not 
using his information on the recurrence classes of the chain {lt)t on this block. In order to 
bound the error due to this approximation, we have to replace the error term based on the 
Li-variation of the process {qt)t>i (which is no more a martingale here) by the Li-variation 
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of an auxiliary martingale representing the beliefs of Player 1 over the recurrence classes of 
the Markov chain {lt)t>i (note however that we obtain a less precise bound since the method 
developed for games with incomplete information does not apply here). Then, we prove that 
the expected value of w{pnToi QnTo) is always greater or equal to w{p, q) using similar tools as 
in [14] and also that w is balanced. We obtain liminf„T;„ > supjiiilii) G yV~(t'), balanced}, 
and by symmetry limsup^iJn < inf{tt;|tt; G W"'"(t)), balanced}. 
Then it remains to show the equality 

sup{t(;|t(; G >V~(t'), balanced} = inf{tt;|tt; G W^(t)), balanced}, 

by reducing the problem to the equality showed by Mertens and Zamir. The conclusion 
follows. 

5 Non revealing games 

Let (T in S be a strategy of Player 1, i.e. a = (aT)r>i, with ■ {K x I x J)^~^ x K — > 
for all T. For each finite history hx in (/ x J)^, we say that Ht is compatible with (p, a) if for 
some r and q we have Pp,q,a,T{hT) > 0. In this case we define pt{p-, (^){hT) in ^{K) as the law 
of the state kx+i knowing that: the initial state ki is selected according to p, Player 1 uses a 
and hx has been played at the first T stages. It does not depend on the last move of Player 
2 in hx, and for all k in K, q in A(L) and strategy r of Player 2 such that -fp,g,(T,r(^T) > 0, 
we have: 

PTiP,Cr)ihT) = Pp^q^aA^T+l = k\hT). 

If hx is not compatible with (p, cr), we define PT{p,(^)ihT) arbitrarily in A{K). For T = 0, 
Po{p, cr) = p is the law of the first state ki. 

Definition 5.1. The relevant information of Player 2 after hj- has been played is: 

PT{p,cr)ihT) = pT{p,cr)ihT)B. 

For each T, we denote by Ht the cr-algebra on the set of plays Q generated by the 
projection on (/ x J)^ giving the first T moves. 

Lemma 5.2. For any strategy pair (o", r) inT, x T, the process {pt{p-,<^))t>q is a {%t)t>o- 
martingale with respect to JPp^q^u^r- 

Proof. Assume (c, r) is played, the initial probabilities on A(K) and A(L) being p and q. Fix 
T > and hx = ■■■,iT,jT) G (/ x J)'^. Given (zT+i,jr+i) in / x J, we write /ir+i = 

(ii, jT, ^T+i, jT+i) and denote by rr(p, cr)(/iT+i) the conditional probability on the state 
kx+i given that hx+i has been played at the first T + 1 stages. We have rxip, o-){hT+i)M = 
PT+i{p,cr){hT+i), and ]Ep^q^a,T{rT{p,cr)\hT) = PT{p,cr){hT)- Now, 

lEp^g^a,T{PT+i{p,cr)\hT) = ]Ep^g^a,T{PT+i{p, cr)B\hT) 

= lEp^g,^ArT{p,a)MB\hT) 
= ]Ep^g^a,T{rT{p,(T)B\hT) 
= PTip,cr){hT)B 

= PT{p,(T){hT). 

□ 
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Definition 5.3. A strategy a in T, is called non revealing atp if for all r and q, the martingale 
{pt{p-,'^))t>q is lPp^q^(T,T-o.lmost surely constant. The set of NR strategies of Player 1 at p is 
denoted 

We will give an alternative definition of non revealing strategies. 

Definition 5.4. 

NR{p) = {x G A{I)^,yi G / s.t. x{p){i) > 0, p{x,i)B=pB} 
= {xe A(/)^, Vi G /, {p''x'{i))keKB = x{p){i)pB}. 

Given a in S and an history /i^ G (/ x J)^ which is compatible with {p, a), for each i in / 
and k in K, we denote by x^{p,a){hT){i) the probability lPp^q^a-,T{iT+i = ^|^T+i = k^hx) that 
Player 1 plays i after hx if the next state is k. This probability does not depend on {q,T). 
The vector x{p,a){hT) = {x^{p,a-){hT){i))k,i is viewed as en element of A(/)^. 

Lemma 5.5. A strategy a in Ti is non revealing at p if and only if for all T > and history 
Ht ^ {I X J)^ compatible with {p,cr), the vector x{p, a) (hx) belongs to NR{pT{p,cr){hT))- 

Proof The proof is similar to the proof of Proposition 6.5 in [20] . Fix p and a. Consider T > 
and a history hx in (/x J)^ compatible with {p, a), and let {q, r) be such that Pp,g,a,T{hT) > 0. 
For simplicity we write x = x{p,a){hT) in A(/)-'^, and pT = PT{p,o'){hT) G A(K). 

Vi G /, Pp^q^aA^T+l = i\hT) = ^ Pp^q^aA^T+l = k\hT)x^ {%) , 

k&K 

k&K 

= x{pT){i)- 

Consider now a history Ht+i = (/it, ^T+ii jT+i) extending hx such that lPp^q^a,T{hT+i) > 0. 
We have {IPp^q^„^r{kT+i = k\hT+i))keK = PTix,iT+i), so pT+i(p, o-)(/iT+i) = p{x,iT+i)M. 
Consequently: 

PT+iip,cr){hT+i) = PTix,iT+i)B. (2) 
We now conclude the proof of the lemma. 

=^ Assume that a is non revealing at p, and consider T and hx in (I x J)-^ such that hx 
is compatible with {p,cr). Write px = PT{p-,'^){hT) and x = x{p, a){hrp). If x{px){iT+i) > 0, 
then (Ht, ir+i) is compatible with {p, a) and by assumption pT+i(P; cr)ihT, ir+i) = Pt{p, cr){hT) 
We have pt+i(p, cj)(/ir, «T+i) = PT(a^, ^T+i)-B by equation (2), and by definition pr(p, o") (/it) = 
PtB. We obtain pT{x,iT+i)B = prB, so x G NR{pt)- 

Assume that for all T > and history hx ^ {I x J)^ compatible with {p,cr), the 
vector x{p,a){hT) belongs to NR{pT{p,(7){hT))- Consider T > 0, and a history /it+i = 
(n, ji, ...,iT+i,jT+i) compatible with {p,a). Write hr = (ii,...jT), Pt = PT{p,cr){hT), and 
x = x{p,a){hT). By equation (2), we have pT+i{p,cr){hT+i) = PT{x,iT+i)B. We have 
x{pT){iT+i) > because hx+i is compatible with (p, cr), so by assumption, f)T{x,iT+i)B = 
PtB, and by definition p{p, a){hT) = PtB. We obtain pT+i{p, cr){hT+i) = Pt{p-, c)(/iT)- D 
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We also have a splitting lemma with non revealing strategies. 

Lemma 5.6. Consider a convex combination p = "^^^g ctsPs in ^{K) having the property 
that psB = pB for all s. Consider for each S a non revealing strategy as in T,{ps). There 
exists a in such that: 

Vg G A(L), Vr G T, Pp,q,a,T = ^ aslPp,,q,as,T- 

The proof is similar to the proof of Lemma 6.6 in [20], with a being defined via the split- 
ting procedure: observe the first state ki in K, then choose s according to the probability 
dsPs^/p^^, and play according to ag- Notice that the equality lPp^q^a,T = Sses c^s-^p^^g^o-^^r is 
valid for any, revealing or non revealing, strategy r of Player 2. See also Lemma 6.6 where 
we study more precise properties of this splitting procedure. 



Non revealing strategies of Player 2 at g are defined similarly, the set of such strategies is 
denoted T{q). 

Definition 5.7. The T-stage non revealing game at {p,q) is the T-stage game where the 
strategy sets are restricted to T,{p) and T{q). It is denoted by TT{p,q)- 

Define Tit{p) as the set of T-stage non revealing strategies of Player 1 at p. Formally, 
Et(p) is the projection of over the (compact) set of T-stage strategies of Player 1. 

Similarly, Triq) denotes the set of T-stage non revealing strategies of Player 2 at g^. The game 
rT(Pi9) can equivalently be seen as the zero-sum game {T,T{p),'TT{q),^x'^)- 

Proposition 5.8. For allT > 1, the T-stage non revealing game at {p,q) has a value denoted 
by VT{p,q)- For each q in A(L) and p* in P* , {p i— )• VT{p,q)) is concave on B~^{p*), and 
similarly for each p in A(i^) and q* in Q* , {q i— )• vt{p, q)) is convex on C^^{q*). 
Moreover, for all T > we have the recursive formula: 

{T + l)vT+i{p,q) = max ^ mm { G{p,q,x,y) + T ^ x{p){i)y{q){j)vTipix,i)M,q{y,j)N) 



xeNR{p) y€NR{q) 



™™ . max ( g, X, y) + T ^ x{p){i)y{q){j)vT{p{x,i)M,q{y,j)N) 



yeNR{q) xeNR{p) 



Proof: Consider a sequence {pa,cra)a>i converging to {p,cr) where da G T,T{Pa) for each 
a. For each history hx in (/ x J)'^ and t in T such that lPp,q,a,T{hT) > 0, the conditional 
iIPpa,q,aa,TikT+i = k\hT))k&K Converges to {Fp^g^^^rikr+i = k\hT))keK, so we can pass to 
the limit in the equality pT{pa,cra)ihT)B = paB to obtain that o" is a T-stage non revealing 
strategy of Player 1 at p. As a consequence, {{p,a),a G is a closed subset of A(Er) x 

Tit{p)-i hence is compact. 

By Lemma 5.6, we obtain that the game {T,T{p),TTiq),'yx'^) is affine-like: for all a', a" G 
TiTip), for all a G [0, 1], there exists a in St(p) such that for all r, 7j,'''((T, r) = a^j^'^{a',T) + 
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Summing up, we can use a minmax theorem due to Fan (1953 see Proposition A. 13 in 
[25]) to obtain that that the T-stage non reveahng game at {p,q) has a value VT{p,q), and 
that both players have optimal strategies. 

Since {{p,a),a G St(j')} is compact, vt{p,q) is upper semi-continuous in p, and by sym- 
metry it is lower semi-continuous in q. Lemma 5.6 gives that VTip, q) is concave in p on each 
set B~^{p*), and convex in q on each set C~^{q*). 

We now prove the recursive formula in a standard way. Fix T > and {p,q) in A{K) x 
A(L). We consider the zero-sum game AT+i{p,q) = {N R{p) , N R{q) , f) , where f(x,y) = 
j^G{p,q,x,y) + ^^i^jj^jx{p){i)y{q){j)vT{p{x,i)M,q{y,j)N). The strategy sets NR{p) 
and NR{q) are convex compact, and one can prove (as in the proof of Proposition 7.4 in [20]) 
that / is concave s.c.s. in x and convex l.s.c. in y. By Sion's minmax Theorem, AT+i{p,q) 
has a value that we denote by aT+iip,q)- 

Consider now an optimal strategy r of Player 2 in the non revealing game TT+i{p,q)- 
Since r is non revealing at q, the first stage strategy ri lies in NR{q). Define x in NR{p) 
which is a best response against y = ri in AT+i{p,q), and a strategy a of Player 1 which: 
plays according to x at stage 1, and from stage 2 on play an optimal strategy in the game 
TT{p{x,i)M,q{y,j)N), where i and j are the moves played at stage 1 (with q{y,j) defined 
arbitrarily if y{q){i) = 0). Using Lemma 5.5, one can show that the strategy a is non 
revealing at p, and the payoff 7j.'''((T, r) is at least f{x,y), so at least aT+i{p-,q)- Since r 
was optimal, we deduce that ax+iip^q) < VT+i{p-,q)- Exchanging the roles of the players, 
we get aT+i{Piq) = VT+i{p,q) and the recursive formula is proved. This ends the proof of 
Proposition 5.8. □ 

We now prove a proposition allowing to consider smaller sets of strategies. 

Definition 5.9. Let T denote the subset of strategies of Player 2 which for all T >0, depend 
at stage T + 1 only on the past history hx ^ {I x J)^ and on the current state It+i- Define 
S similarly. 

Proposition 5.10. For all {p, q) G A(i^) x A(L), for all t ^T, there exists a strategy T £ T 
such that for all T > 1 and o" G S, 7^''^(o", r) = 7^'''(cr, r). Moreover, if t is non-revealing at 
q, then r is also non-revealing at q. 

As a corollary, in Txip, q) and in the non revealing game Trip, q) the players have optimal 
strategies in S and T ■ 

Although it is intuitive that Player 2 does not need to remember the whole sequence of 
states (/i, /t, ...), the proof of the proposition requires to be precise. 

Proof of Proposition 5.10. The strategy r proceeds as follows: for t > 0, at stage t + 1, Player 
2 does not remind the past states (/i, .., Zj), but using the history ht, he can generate a virtual 
sequence (Ii, ..,It) using the conditional law of (Zi, ..,/t) given {ht,lt+i) under JPp.q^a.r (which 
does not depend on a). He selects then at stage t + 1 an action jt+i with the probability 
T{ht,li, ..,lt,k+i)- Formally, we have 

T{ht){lt+l) = lEp^q^„^r[T{ht, h, k, k+l)\ht, k+l]- 
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We will prove that for all t > 1, (/it, k, h) has the same distribution under lPp^q^a,T and under 
^p,q,cr,T, i-e. for all h G Hf, for all (A;, I) G K x L, 

F'p,q,a,T{ht = h,lt = I, kt = k)= lPp,q,cT,T{^t = h,lt = I, h = k). (3) 

Let us proceed by induction on t. For t = 1, the property is obviously true since ti = ti. 
Assume that the property is true for f > 1. At first, the conditional distribution of It+i given 
{ht,kt,k) is M{lt,.) under Pp^q.a,T and under Pp^q^^j-f. We deduce that the law of {ht,lt+i) 
is the same under both probabilities. By construction the conditional law of jt+i given 
{ht, k+i) is the same under Pp^q^a,T and under Pp^q^a,T-, which implies that the conditional law 
of (/f+i, jt+i) given ht is also the same. Using that (/c^+i, it+i) and {lt+i-,3t+i) are conditionally 
independent given ht (under both distributions) and that the conditional law of {kt+i,it+i) 
given ht does not depend on r, the conditional distribution of {kt+i,lt+i^,it+i-, jt+i) given ht 
is the same under Pp^q^a,T and under Pp^q^^j-f. We deduce that the law of {htJ^i^kt+i-,h+i) is 
the same under both probability distributions which concludes the proof of (3). Going back 
to the main proof, we obtain 

T-l 

7f,'''(cr,T) = ^Ep^q^„^^[Pp^q^„^r[g{kt+l,k+l,H+l,it+l)\M] 
T-l 

= ^ ^p,q,a,T[Pp,q,a,T[9ikt+l,k+l,k+l,jt+l)\ht]] 
t=0 

Moreover, since we proved that the law of {ht,lt+i) is the same under Pp^q^a,T and under 
Pp,q,<j,Ti this implies that qt{Q,'r) and qt{q,T) also have the same law, which proves the last 
assertion. 

Remark 5.11. Note that in the above construction, the laws Pp^q^a,T and Pp^q^a,- do not 
coincide on the whole set Q but generate the same payoffs. The reader may convince himself 
by considering the following example: 

^ = {1, 2}, iV = (^/^ J/^) , q = (1/2, 1/2), J = {1, 2}, 

and r is defined by: play 1 at stage 1 and then at stage t > 2, play 1 if It = lt~i and 2 
otherwise. A direct computation leads to 

lPp,q,aAh = k = 1,32 = 1) = 1/6, Pp,q,a,r{h = h = IJ2 = I) = 1/18. 

Remark 5.12. In the above proposition, we made use of some standard properties of Pp^q^a,T, 
namely for all t > 1: the conditional law of (/i, .., It) given (ht, h+i) does not depend on a, the 
conditional law of {kt+i,it+i) given ht does not depend on r, {kt+i,it+i) and {lt+i,jt+i) are 
conditionally independent given ht- All these properties can be obtained by direct algebraic 
manipulations, the details are left to the reader. 

The next proposition is a major point of our proof. 

Proposition 5.13. For all T > 1, vt is 3-Lipschitz on A{K) x A(L). 
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The proof of Proposition 5.13 being quite technical, it is postponed to section 7. The 
crucial consequence is to obtain uniform equi-continuity of the family of functions {vt)t>i- 
In Renault [20], the functions {vt)t>i where only proved to be u.s.c. and this property would 
not have been sufficient here, since the proof of our main Theorem 6.1 relies on uniform con- 
vergence of the ('Ur)T>i to their limit. The main point is to introduce a function S{p,p') (see 
Definition 7.2) which will play the role of a metric and with respect to which all the functions 
{vt)t>i are "1-Lipschitz" . However, we do not know if the functions {vt)t>i are 1-Lipschitz 
with respect to ||.|| or if there exists a real metric having this property. 

Before proving that the non revealing values vt converge when T goes to infinity, we now 
present a definition and a lemma, allowing to concatenate non revealing strategies defined on 
blocks. 

Definition 5.14. Given Tq,Ti > 1 and a strategy cr G S such that for all 1 < t < Ti, 

(JTo+t does not depend on {ki, ...^kxo), we define for all Htq G a Ti-stage strategy a{hTo)- 
This strategy a{hTo) plo-ys, after a history (/i^-i; k'l, /cQ, what a plays at stage Tq + 1 after 
{{ho, k[, k't). Formally, 

o{hTMK-iA, ■■■A) = <yTM{hnX-i)A, <t <Tiyh't e Ht-iy{k^ e k' 

Lemma 5.15. Consider Tq,Ti G M* and strategies (fx, r) G SxT such that for all 1 < t < Ti, 

CTTo+t 0-nd ttq+i do not depend on, respectively, {ki, /ctq) CLiT'd {h, ■■Jtq)- The conditional law 
of {kTo+t,lTo+t,iTo+t, jTo+t)t=i,..,Ti given /itq under Pp^^^^^r is precisely ]Pp^^^q^^^„^hTo),r{hTo)- 

The proof follows easily from a direct computation. 

Remark 5.16. Note that ifr^T, then the above condition on r holds for all Tq,Ti. We 
will use the following consequences in the sequel, at first 

Ti 

lEp^q,a,T[Yl 9{kTo+t, ho+t, iTo+tjTo+t)] = lEp^q,a,TbT\'^'^'^° (^^(H ) > '^(H ))] > 
t=l 

and for all t = 1, and any continuous function f defined on A(Er) x A(L), for all hxo 

in {I X J)^o, 

lEp,g,CT,T[f{PTo+t, qTo+t)\hTo] = ^PTo,qTo,'7{hTo),r{hTo)[f {Pt, Qt)], 

where ptq = PTo{p,(^){hTo) and qr^ = qTo{q,T){hTo). 



Proposition 5.17. v{p,q) = liinT~^oDVT{p,q) exists and is balanced. 

Proof Let {p,q) G A{K) x A(L), p* = pB, q* = qC and e > 0. Choose To,Ti G M* such 
that 

\\M^°-B\\<e, \\N^^^-C\\<e, ^ < e, 
and vti{p*,Q*) > limsupj^v-rCp*; 9*) ~ ^• 
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Given G IN* and an optimal strategy r G T of Player 2 in the game f Ar(To+ri)(j') q)j let us 
construct the strategy a as follows. For n = to — 1, during stages t = n(To + Ti) + 1 to 
t = n(To + Ti) + To, play a fixed action io G /. During the next Ti stages, play an optimal 
strategy in the game f Ti (p„(To+Ti)+To (P, o"), 9n{To+Ti)+To(9: ^))- The payoff can be written as 

N-l / n{To+Ti)+To 

iV(ro + ri)7^^y^^^^)(cr,T) = ^ ^ lEp^q,a,T[9{km,lm,imJm)] 

n=0 ym=n{To+Ti)+l 

{n+l)(To+Ti) 

m=n(To+Ti)+To+l 
N-l 

\ ( rr , rr r Pn(To+Ti){P:'^).'?n(To+Ti) (q.-^) / /, N /, 

> 2^ (^-To + T^liEp,<^,<x,r[7Ti (<^(^n(To+Ti)),^(/in(To+Ti 
n=0 



> X] (-^0 +TiiE;p,<;,<^,^[z;Ti(p„(To+Ti)+To(P,0-),9n(ro+ri)+To(9,T))]) 
n=0 

AT-l 

n{To+Ti) 

n=0 

> J] {-To + Ti{vT^{p*,q*)-Qe)). 



n=0 

The first inequality follows from Lemma 5.15 and the fact that g is valued in [—1,1], the 
second follows from the definition of a. Note then that 

PniTo+Ti)+To {P^ = Pn(To+Ti) (P, f^)M^o 

since the strategy a is independent of the state variable during the block of size Tq and that 

^p,q.'yA^n(To+Ti)+To{l->'^)\^n(To+Ti)\ = Qn{To+Ti) (g,T)iV^o. 

The third inequality is obtained by taking conditional expectation with respect to h^t^Xg^Xi) 
and using that vt^ is convex with respect to the second variable on C~^{q*). The last inequal- 
ity follows directly from the properties of Tq and the fact that vti is 3-Lipschitz. We deduce 
that for all N e JN* 

^'7V(To+Tl)(^',9) > 1 . >vtAP :Q )-8e> limsuprVTiP ,q ) - 9e. 

It follows that 

limmfNVN(^To+Ti)iP, q) > lm\suprpVT{p* ,q*) - 9e. 
Using that Hvt — ^t+T' I loo < this implies 

liminfT'0T(P5 ^z) > limsupyi;T(p*, (7*) — 9e 

and finally liminfT{jj'(p, > limsupq^VTip* , q*)- Inverting the role of K and L leads to 
limsupj-v-rCp, ^ limmiTVT{p* ,Q*), and we conclude that v{p.,q) = \\Ti\T-^ooVT{p-,q) = 
lim.'TVTip* ,q*) exists and is a balanced function. □ 
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6 Back to the original game and the Mertens-Zamir 
system 

Recall that W denotes the set of continuous functions w : A(Er) x A{L) — > [—1, 1]. 
Theorem 6.1. v = Iiuitvt exists and can be characterized by: 



inf 



sup 



w G W, w is I-concave, balanced and 
\/{p,q) G A(K) X A(L), w{p,q) > vexiiMax{w,v){p,q) 

w G W, w is Il-convex, balanced and 
V(p, g) G A(ir) X A(L), w{p,q) < caviMm{w,v){p,q) 



Before we prove Theorem 6.1, we will prove a proposition and two lemmas. We first show 
that the comparison result of Mertens and Zamir can be extended to our case. 



Proposition 6.2. If we define 
V = inf 



w G W, w is I-concave, balanced and 
y{p,q) G A(i^) X A(L), w{p,q) > vexnM&x{w,v){p,q) j ' 



V = sup 

then V = V. 



w G W, w is Il-convex, balanced and 
y{p,q) G A{K) X A(L), w{p,q) < caviMm{w,v){p,q) j ' 



Proof. Consider any balanced function w in W. One may identify w with its restriction w* 
to the product of simplices P* x Q*. I-concavity and //-convexity on A{K) x A(L) of w is 
equivalent to /-concavity and //-convexity of w* on P* x Q* (using that B and C are linear). 
We will prove that cav i{w) is balanced and that (cav/it;)* = cav i{w*). 

Fix {p, q) be in A{K) x A(L), and define = pB and g* = g/J. Recall that 

N N N 

caviw{p,q) = sup{^ amw{pm,q) \ N > l,\/m = l,..,N, a™, > 0, ^ = 1, ^ amPm = 

m=l m=l m=l 

Given {am,Pm)m.=i,..,N as above, for all m, w{pm,q) = w{pmB,q*), and J2m=i^rnPmB =p*, 
which implies 

TV N 

amw{pm,q) = ^ amw{pmB,q*) < cav/(7i;*)(p*,g*), 

m=l 771=1 

and therefore cavjw{p,q) < cavj{'w*){p* ,q*). 

Define now the affine map / from P* to A{K), depending on p, by: 

Vp'GP*,Vr = l,..,rM,VA;GK(r), /(p')'=|E'^'"('^^!2 ^^f'^f S'^'"^'^ ^ ^ • 
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We have f{p*) = p. Moreover, for any recurrence class K{r), we have ^^ZkeKir) f(p')'' ~ 
Y1is&k{t)P"^ ■ This imphes that for all p' G P*, f{p')B = p' . Consider now {a.m-,p*m)m=i,..,N 
such that 

N N 

Mm = 1, ...,iV, am > 0,P*m S P* , ^ = 1, ^ amP*m = P* ■ 

m=l 771=1 

We have by construction "^^=1 ^mf{Pm) — P ^'^d w{pm,q*) = w{f{pm),q) which implies 

N N 

^ amw{pl^,q*) = ^ arnw{f{pl,),q) < caviw{p,q), 

m=l 771=1 

and therefore cscvi{w*){p* ,q*) < cmiw{p,q). We conclude that: 

y{p,q) G ^{K) X A(L), cmiw{p,q) = cm i{w*){p* ,q*). 

Consequently, cav7(w) is balanced and (cav/tf)* = cav/(t(;*). 
Since v is also balanced, we deduce that : 



sup 



w : P* X Q* — > [—1, l],w continuous //-convex and 

y{p*,q*) € P* X Q*, w{p*,q*) < csviMm{w,v){p* ,q*) 



and a similar property for v* . The equality v* = v* follows from Theorem 2.1 in [14] and 
implies v = v- □ 

Remark 6.3. The above proof relies heavily on the fact that B is associated to a recurrent 
Markov chain. For example, recall example E of section 4-' 

1 \ / 1 0> 

M=|0 1 0,B=0 1 
^1/2 1/4 1/4/ \2/3 1/3 Oy 

In this case, the function f{p) = (p^ — 1/3)"'' (where ^ denotes the positive part) is such that 
cav/(po) = and cav(/*)(po^) = 2/9 for po = (0,0,1). This was already pointed out in 
Renault [20], where general Markov matrices were considered. Our reduction of the problem 
to the study of aperiodic recurrent Markov chains allows therefore for useful technical simpli- 
fications. For example, in case L is reduced to a single point, our characterization becomes 
v{p) = cm{v){p). Using aperiodic but possibly not recurrent Markov chains, the characteri- 
zation given in [20] reduces to v{p) = cav{v){pB) where v and B are defined similarly. One 
checks easily that these two expressions coincide. 

Remark 6.4. Note that one may add the condition that w is also I-concave in the supremum 
defining v, because if w fulfills the required assumptions, then cav/w also and cavjw > w. A 
similar result holds for v. 

Lemma 6.5. Let w £ W be such thaty{p,q) G A(/(') x A(/y), w{p,q) < cav[Mm{w,ij){p,q). 

If w{p,q) > v{p,q), then there exists a convex combination {am,Pm)m=i,..,K in [0,1] x 
A(/^) such that 

K K K 

^ am = 1, P = ^ OlmPm-, Vm = 1, .., K, w{pm, q) < v{pm, q), and ^ amw{pm, q) > w{p, 

m=l m=l m=l 
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Proof. Using Caratheodory's Theorem, there exists {am,Pm)m=i,..,K in [0,1] x A{K) such 
that 

K K K 

^ = 1, P = ^ OimPm, ^ arrMm.{w ,v){j)rn-, O) = C&Y lMm{w ,v){p, q) . 
m=l m=l m=l 

This imphes that for ah m = 1, .., ii', 

Mm{w,v){pm,q) = ca\iMm{w,v){pm,q) > w{pm,q), 
so that Mm{w,v){pm, q) = w{jpm-,q) < v{jpm-,q)- To conclude, note that 

K 

amw{pm,q) = caviMm{w,v){p,q) >w{p,q). 

m=l 

□ 

Lemma 6.6. Consider a convex combination p = ^^ggOsPs in ^(K). Consider for each 
s G S a non revealing strategy ag in There exists a in such that: 

Vg G A(L), Vr G T, Pp,q,a,T = ^ as-ff'p,,g,c.,,r- 

SG5 

As a consequence we have for all T G IN* , q in A(L) and t G T, 

T-1 T-1 
lEp^q,a,T^^ \\qt+i{q.T) - qt{q,T)\\] = J] asiBp^,g,^^,^[^ Ut+iiq^r) - qt{q,T)\\]. 

t=Q s€S t=0 

Moreover, for any function w £ W which is I -concave, II -convex we have 

Mt > 1, IEp^q^„^r[w{pt,qt)] > y^^asw{psB,qC). 

Proof. Let us define a as in Lemma 5.6 via the sphtting procedure: observe the first state ki 
in K, then choose a variable s in 5 according to the probability UgPg^ /p^^ , and play according 
to as- It defines a probability IP on 0,x S and using Kuhn's Theorem, there exists a such that 
the induced probability on Q is equal to IPp^q^a,T- Moreover, for any event A in Q, we have 
IP{A\.s) = IPp^^g^f^_.^T-{A) almost surely, which implies the equality IPp^q,a,T = Z^sgs '^s-^ps ,9,0-3, t- 
Recall that whenever ht is compatible with (g, r), qt{q,T){ht) does not depend on {p,cr). 
Since qt{q,T) = Qt{Q,T)B, the preceding equality implies that 

T-1 T-1 
IEp^g,aAj2 - =^(^sIEp,,g,„,,r[J2 Mt+iiQ, r) - qt{q, t)\\]. 

t=0 s€S t=0 

Let us now work with the probability IP. Since ag is non-revealing at ps, we have for all 
stage i, s in S" and history ht: 

{IP{kt+i = k\ht,s))keKB = {IPp„q,as,r{kt+i = k\ht))keKB = pt{Ps,crs){ht) =PsB. 
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Denote by IE the expectation under IP, and by s the random variable with values in S of the 
first choice of Player 1. The above line implies: 

]E[psB\ht] = lE[{lP{kt+i = k\ht,§))keK\ht]B = pt{p,a){ht)B = pt{p,a){ht), 

Using that w is /-concave and Jensen's inequality, we deduce that 

lE[w{psB,qt{q,T))\ht] < lE[w{pt{p,a),qt{q,T))\ht]. 

Using again that qt{q,T) does not depend on {p,cr), we obtain 

lE[w{psB,qt{q,T))] = lE[lE[w{psB,qt{q,T))\s]] = ^asJEp^,q,„„r[w{psB ,qt{q,T)]. 

To conclude, w being //-convex, and using Jensen's inequality, 

Vs G S, IEp^^q^„^^r[w{psB ,qt{q,T)] > w{psB,qC). 

□ 

Let us now turn to the proof of the main theorem, making formal the description given 
in the end of section 4. 

Proof of Theorem 6. 1 . 

(1) Fixing norms. Recall that if a; = {xs)s^s is an element of an euclidean space JR^ , \\x\\ = 
X^sgsI^^I- ^ -^*' °^ strategies St is the set of maps from uJSq{K x / x 
J)*^^ X K to A(/) which is seen as a subset of . can be seen as a compact subset 
of (i?i^)^t=o {Kxly.jy ^xK gnJowed with the corresponding norm. Note at first that for all 
T G IN*, for aU {p, q) G A{K) x A(L), and for aU a G St and r, r' G Tt, 

|7r(^,r)-7r('^,r')|<||r-r'||. 

(2) Here we show that we can approximate strategies with low variations of the martingales 
Pt and qt by non revealing strategies. Let T be fixed. For j > e > 0, define T,j, as the set of 
strategies a G St such that at any stage and for any history of the game, all the actions are 
played with probability at least e. Formally, 

yt = l,..,T,yht-i G Ht-i,y{ki,...,kt^i,k) G K\yi G I,at{ht^i,ki,..,kt.i)''{i) > e. 

Define the map from = A{K) x A{L) x ^^j, x Tt to if by 

Feip,q,a,T) = {lPp^g^„^riht){qt+iiq,r){ht) - ^)(/it-i)))t=o,..,T-i , hte{/x J)* ' 

where for ht = (ii, Ji, ••, "itiit), /it-i is defined here as {ii, ji, ..,it-i, jt-i), and T is the large 
but finite integer T = LY^J~q{I x J)*. Note that F is continuous and that Rg is compact. 
Define the compact set Rg = Ff^{{0}), it is the subset of Rs consisting of elements {p, q, a, r) 
such that T is non-revealing at q. This follows easily from the observation that if an history 
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ht is compatible with {q,T), then due to the definition of we have Fp,q,a,T{ht) > for all 
p and a G Note also that by construction 

T-l 

\\F(.P,Q,cr,T)\\ = ^ lEp^q^^^r[\\qt+i(.q,r) - qt{q,T)\\]. 
t=0 

Claim: For all £ > and T e IN*, there exists Ci = Ci{T,e) such that 

V(p,9,o-,r) G Re, 3{'p,q,a,f) G -R^, ||(p,g,(7,r) - {p,q,a,f)\\ < e + Ci\\F{p,q,a,T)\\. 

To prove this claim, cissume by contraciiction that for all n G ^V*, tliGrc exists (p^i, Qn-) ^n-^ ^n) ^ 
such that for ah {p,q,a,f) G Re, \\{Pn,qn,crn,Tn) - ip,q,a,f)\\ > e + n||F(p„, c7„, r„)||. 
By compactness, we can extract a convergent subsequence with limit {p* ,q* ,a* ,t*) such that 
F{p*,q*,a*,T*) = and ||(p*,g*,cr*,r*) — {p,q,a,f)\\ > e. which is a contradiction. 

(3) Here we show that an optimal strategy of Player 1 in a nan revealing game TtIp, q) is 
good against strategies of Player 2 that are non revealing at q' , where q' is close to q. 

Claim: for all e > and T G IN* there exists C2 = C2(T,£) such that for all {p,q) G 
A(K) X A(L), for all a G St(p) which is optimal in Txip, q), for for all q' G A(L), and for all 
r' G Uq'), 

7^\a,T') >-e- C2\\q - q'W + Vrip^q)- 

For a proof, assume by contradiction that for all n G IN* , there exists qn) G A{K) x 
A(L), Gn G TiTiPn) which is optimal in TT{pn-,qn)-, q'n £ A(L), and r^'^ G Ti^q'^) such that 

7f,"'''"(cr„, r^) < -£ - n||g„ - q'J\ + vxiPn, qn)- 

By compactness, there exists a convergent subsequence with limit {p* ,q* ,(j* , q'* , r'*). Since the 
left hand-side of the above inequality is bounded below by —1, we have q* = q'* . Moreover, the 
sets of non-revealing strategies being closed, a* is non-revealing at p* and r* is non-revealing 
at q*. Using step (1), we have for all n, 

VTiPn.qn) - - T„|| < 7?'"''"K,T*) " \\t* - T„|| < 7^"'^"(a„,T;). 

It follows that 

VT{p*,q*) < -£ + VT{p*,q*), 

a contradiction. 

(4) Here we control the error while perturbing an optimal strategy in T(p, q) by a non revealing 
completely mixed strategy. 

Let denote the strategy in St which plays an uniform distribution over / at all stages 
independently of the history of the game, notice that cr^ is non-revealing at all points in 
A{K). For all {p,q) G A.{K) x A(L), let crx{p,q) denote an optimal strategy of Player 1 in 
T{p,q). Denote by o'T,eip,q) the strategy which plays: cr^ with probability e, and (Jrip^q) 
with probability 1 — e. The strategy aT,eiP, q) is non revealing at p, and we have for all r G Tt, 
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Using step (2), for all {p,q) G A{K) x A(L), and all t eTr, 

^{p,Q,^,t) G Re, \\{p,q,(yT,e(.P^q),r) - {p,q,a,f)\\ <e + C2{T, e)\\F{p, q, aT,e{p, q),T)\\. 
Using then (1) and (3), 

ItH^^tAp^ q)^'T)= ^7t''(^t> "t) + (1 - £)1t'^{(^t{p, q),r) 

> -2e- ||r-f|| +7T'^{aT{p,q),f) 

> -2e-\\T-f\\-e-C2iT,e)\\q-q\\+VTip,q) 

> z)t(p, g) - 3e - (1 + C2(r, e))C, (T, e)\\F{p, q, aT,e {p, q),T)\\. 

(5) Here we prove that: liminfru-r > v. 

Let w £ W he a balanced /-concave, //-convex function such that 

y{jp,q) G A(/C) X A(L), w{p,q) < caviMm{w,v){p, q). 

Fix {p,q) G A(/C) x A(L), we have to show that limmixVTip,q) ^ w{p,q). 

Let e > and choose Tq G /V* such that Huyg — uHoo < £• For simplicity, we will write pt 
for pt{p,cr), qt for qt{q,T) and Ci,C2 for Ci(To,e), C2(Toe). Given G /V* and an optimal 
strategy r G T of Player 2 in T^Toip, q), we define a strategy a as follows. 

The set of stages is divided into consecutive blocks of length Tq. For n > 0, at the 
beginning of block n, i.e. at the beginning of stage tlTq + 1, Player 1 determines his strategy 
for the block according to the random variables pnTo aiid 9nTo- Two cases may occur. 

If v{PnTo,qnTo) ^ ^ (PnTo ; 9raTo ) ; then (T play during the next Tq stages the non revealing 
strategy axoAPnTo, qnTo)- 

If v{PnTo,qnTo) < ^ (PnTo j 9nTo ) j using Lemma 6.5, there exists a convex combination 
, Pn,m)m=l,..,K in [0, 1] X A{K) which depends on {pnTo, q-nTo), such that 

K K 

an,m = 1, PnTo = X] «",mPn,m,Vm = 1,..,K, w{pn,m,qnTo) < v{Pn,m,qnTo) 
m=l m=l 

K 

and ^ an,m?i'(/'n,m, qnTo) > w{PnTo , ^nTo)- 
m=l 

In this case Player 1 plays at block n the strategy o'To,e{PnTo,qnTo) S S^o given by Lemma 
6.6 and such that for all r' G Ttq, 

K+l 

^PnTo,<lnTQ,'^Tg,e:{PnTg,qnTo),T' = '^n,mPp„^^ ,q„T^ ,(rT,e: {Pn,m ,qnTQ) ,r' ■ 

m=l 

This ends the definition of o". 
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We have to show that the payoff 7ArTo (c, r) is large. It can be written as 

^ N-l (n+l)To 

lNTo{cr,T) = j^"^ ]Ep^g^^^r[ Yl ^P,q,<rA9ih,lt,it,jt)\hnTo]] 
^ n=0 t=nTo+l 



n=0 



where T{hnTo) ^ '^o '^{^nTo) S T,Xq are the continuation strategies defined in Definition 
5.14. 

Claim: For all n = 0, .., — 1 and hnTo occurring with positive probability, 

1tT^°''^"^°{(^{KTo),t{Kto)) > w{PnTo,(lnTo) - 3e - (1 + C2)Ci ||F(p„ro , QnTo , 0-(^nTo ) , T"(/l„To ) || • 

Proof of the claim. For the case v{pnTo, QnTo) > w{pnTo,QnTo), we have (j{hnTo) = (^To,e{PnTo,QnTo) 
and 

^PnTo,QnTo (aXo ,e{PnTo , QnTo) , T (Kto)) > VToiPnTo, QnTo) - 3e - (1 + C2)Ci\\F {pnTo , QnTo , Cr{hnTo) , ^ihnTo) 

> w{pnTo,(lnTo) - 3e - {I + C2)Ci\\F {pnTo, InTo: (^{Kto) ,T{hnTo)\\- 

For the case viPnTo,QnTo) < w{pnTo,qnTo), we have a{hnTo) = o'To,eiPnTo, QnTo) and 

K 

7rr°''^"^°(o'To,e(PnTo,gnTo),T(/lnTo)) = ^ (^n,mj^^° '''"'^^ {<yT,e{Pn,m, QnTo) , T{hnTo)) 

m=l 

K K 

- Y '^n,mVToiPn,m, QnTo) - 3e - (1 + C2)Ci ^ an,m\\F {pn,m, QnTo , (^T,e{Pn,m, QnTo) , t{KTo) 
m=l m=l 

> w{{pnTo,QnTo) - 3e - (1 + C2)Ci\\F {pnTo , qnTo,0'TAPnTo, QnTo)) , t(.Kto)\\, 

where the last inequality follows from the Lemmas 6.5 and 6.6. □ 
Let us now prove that: 

Vn = 1, ..,N - 1, lEp^g^^^r[w{pnTo,qnTo)] > w{p,q). (4) 
By induction, it is sufficient to prove that for n = 0, .., — 2, 

^p,q,a,T[^iP(n+l)TQ,q(n+l)To)\hnTo\ > W {pnTo , qnTo) ■ 

Let us at first consider the case v{pnTQ-,qnTo) > wipnTojqnTo)- Since a{hnTo) is non-revealing 
at PnTo; we have P(n+i)To = PnTo- Then, using that w is balanced, it is sufficient to prove that 



^p,q,a,T[w{PnToB,q(^n+l)ToB)\hnTo] > w(p„To^, ^nTo-B) 
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The process qt = QtB being a martingale, the above inequality follows from the fact that w is 
//-convex and Jensen's inequality. Let us now consider the case v{pnTo, QnTo) < 'w{pnTo,qnTo)- 
Using Lemma 6.6, we have 

^p,q,a,T [w^(P(n+l)To 9(n+l)To ^) I ^uTq] 

= ^PuTo .QnTo ,^T,e (PnTg ,qnTo) AhnTg ) (^^0 (PnTo , ^T,e {PnTo , QnTo )) , QTo (^nTo , r{hnTo ) )] 
K K 

> ^ 0!n,m'w{pn,mB , QnToB) = ^ an,mW{pn,m, QnTo) > w{pnTo-, QnTo) ■ 
m=l m=l 

This proves the above inequalities (4). 
Summing up, we have: 

7AfTo(o-,r) > W{p,q) - 3e - (1 + C2)Ci— ^ ]Ep^q^„^r[\\F{pnTo,qnToi(^{hnTo),r{hnTo)\\- 

n=0 

Finally, using Lemma 5.15, and the classical bound on the Li-variation of martingales (see 
for instance Proposition 3.8. in [29]), we obtain: 

N~l N-l (n+l)ro 

'^JEp^q^„^-r[\\F{pnTo,qnTo,CF{hnTo),T{KTo)\W= lEp^q,a,T[IEp,q,'^A X] " 

— (lt-l\\\hnTo\\ 

n=0 n=0 t=nTQ+l 

NTo 

= ^ ^p,q,<rAMt - qt-i\\] 

t=l 

< Vivro(L-i). 

We conclude that: 



77VTo(<T,r) > w{p,q) - 3e - {1 + 02)01^^^^^ 



Since r is optimal in TiyToiP^ q)^ we deduce that 

VNTo{p,q) > w{p,q) -3e- (1 + C2)Ci ^\ 

V jV 

and then 

hmmfiyVNTo{p,Q) > w{p,q) - 3e. 
Using that Hit — vt+t'\\oo < ^1 this implies 

\\m.miTVT{jP,q) > w{p,q) — 3e, 

and finally liminf ^^^(p, g) > w{p,q). We deduce that limmfTVT{p,q) > v- 

(6) We conclude the proof of Theorem 6.1. Symmetrically, we obtain as in point (5) that 
limsup2^VT(p, 9) < V. The conclusion follows therefore from Proposition 6.2. □ 
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7 Proof of Proposition 5.13 



Let us begin with a few definitions. Recall that K is partitioned into recurrence classes 
K{1), K{rM), each recurrent class r having a unique invariant measure p*{r) G A{K{r)). 
P* = conv{p*(l), ...,p*{M)} is the set of invariant measures for M, as well as for B = lim^M*. 
In the following we denote by A{rM) the set of probabilities over the recurrences classes of 
M, i.e. over the finite set {1, ...^vm}- 

Notation 7.1. Let p be in A.{K). 

For each r, we denote by X{pY = '^k£K{r)P^ probability of the recurrence class r 
under p. And we define the probability X{p) £ A{rM) by \{p) = {X{pY)r=i,...,rM- 

For each r = l,..,rM, we denote the conditional law, under p, of k given r as the vector 
P|r G ^{K) such that: 

V\r = 

It follows that for all p G A(i^'), p = YZ-=i X{pYp\r- Since p^^ = for k ^ K{r), p^^ will be 
often assimilated to a vector in A{K{r)). 

Definition 7.2. The map S : A{K) x A{K) M+ is defined by: 

yp,p'GA{K), s{p,p') = \\x(j))-x{p')\\+ Yl Kp'Y \\p\r-p\r\\- 

The asymmetric map S will play an important role in the sequel, and will be used as a 
(quasi, semi-) metric. S is clearly not a metric, but separates points (we have S{p,p') = if 
and only if p = p') and enjoys the following useful properties. 

Lemma 7.3. For all p,p' G A{K), 

\\p-p'\\ < S{p,p') < 3\\p-p'l S{pM,p'M) < S{p,p'), 

S{pB,p'B) = \\pB-pB\\ = ||A(p) - A(y)||, and S{p,p') = \\p-p\\ if pB=pB. 



^ if X{pY >0,kGK{r}, 
p*{rf if X{pY = 0, ke K{r), 
if k4 Kir). 



Before proving the lemma, we will present the roadmap of the proof 



Definition 7.4. 



1) For a G [0,1], define the operator on the set W of continuous functions f defined on 
A{K) X A(L) by 

^MYp^i) = max ^ min ^ ( a G{p,q,x,y) + (1 - a) ^ x{p)(i)y{q){j)f{p{x,i)M,q{y,j)N) 



xeNR{p) y€NR{q) 
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2) Let T he the subset of functions f in W satisfying: 

. \fp,p' G A{K), \fq G A(L), f{p,q)-f{p',q) < S{p,p'), 
. Vp G A{K), Vg, g' G A(L), /(p, g) - f{p, q') < S{q', q), 
• \fp*,q* G P* X Q*, / is I-concave and Il-convex on B~^{p*) x C^^{q*). 



We now present our strategy for the proof of Proposition 5.13. We have vt+i = ^ i (vt) 

t+i 

for t > (with the convention vq = 0), and all the functions in are 3-Lipschitz using Lemma 
7.3. As a consequence, to show that vt is 3-Lipschitz for each t, it is sufficient to show that 
for all a, J- is stable by ^a- 

To do so, the main point is the following: given p and p' in A{K) and x in NR{p), one 
need to find p' in NR(p') such that (p', x') is not too far from (p, x). If p and p' belong to the 
same set B~^{p*), i.e. if p and p' assigns the same weight to each recurrence class, this can 
be efficiently done using, within each simplex A{K(r)), a lemma by Laraki ([11], see here the 
Appendix at the end). If pB ^ p'B, there exists a simple affine map to transfer probabilities 
in B~^{pB) to probabilities in B~^{p'B) (Lemma 7.12). In the general case, we need to 
combine both aspects, applying first the affine transformation then the Laraki splitting (see 
Lemma 7.13). In the computations, the expression S{p,p') appears and allows to control the 
expressions. 

We now prove Lemma 7.3. After this, all the rest of this section is devoted to the proof 
that: 

Va G [0,1], $„(J') C T. 



Proof of Lemma 7.3. 

\\p-p'\\= E E IKpYPir-Kp'Yp'fl 

+ E E p"+ E Ep' 

r ■ X{pY=0 , A(p')''>0 keK{r) r : A{p)''>0 , X{p'y=0 k£K{r) 

< E E {iKprpfr-KprpU + wyptr-Kp'Yp'fi 

r:X{pYX(p'y>0 k€K{r) 

kp'y + E ^^py 

r:X{pY>0,X{p'y=0 



+ 


E 


r : X{py 


=0, A(p')''>0 






Eiw 

r=l 


-Kpr\ + 







Jk\ 



r: X{pYX{p'Y>0 keK{r) 
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On the other hand, 



S{p,p') = \\Xip) - X{p')\\ + HPr\\P\r-p\r\\ 

<\\x{p)-x{p')\\+ Yl E {WYPir-p'\ + \p'-p"'\ 

r:\{pyX{p'y>0 keK{r) 

= \\xip) - x{p')\\ + Yl (\Kpr-HpY\+ E \p'-p"\ 

r: \{pyX{p'y>0 \ keK{r) 

< 2\\X{p) - X{p')\\ + IIp-p'II < 3\\p-p'\\. 

For the next inequahty, note that X{pM) = X{p) and {pM)^^. = p\rM, so that using that 
p — )• pM is non-expansive 

S{pM,p'M) = \\Xip) - X{p')\\ + Y Kp'MP\r-p\r)M\\ 

r:X{py-X{p'y>0 

< \\Xip) - X{p')\\ + Y Kp'mP\r-p\r)\\ = Sip,p'). 

r:X{pyX(p'Y>0 

The two last equahties follow easily by a direct computation. □ 



Fix p in A[K) and q in A(L), and let p* = pB and q* = qC . To any x S A(/) , one 
associates z G /S.{K x /) by z{k,i) = p''x^{i) so that p{z,i) = p{x,i). We will need a more 
abstract way to denote conditional probabilities. Let Af{A(K)) denote the set of probabilities 
with finite support on A{K). Such a probability can be written as 

N 
n=l 

with {an,Pn)n=i,..,N S ([0,1] X A{K))'^ such that J2n=i'^n = 1 and p = J2n=l,..,N °^nPn- 
This set is endowed with the usual weak* topology which is in particular metrized by the 
Wasserstein (Kantorovich- Rubinstein) distance induced by the norm ||.|| and denoted dw- 
Recall that 

V/i, G A{f{A{K)), dw{ij,, z^) = max{ J (pdfi — J ^'dv \ (p € Lipi}, 

where Lipi denotes the set of 1-Lipschitz functions on A{K). We will use this particular 
metric on Af{A{K)). Let us also recall the classical duality theorem (see e.g. Theorem 1.14 
in [28]) {Kantorovich duality formula): 

\/li,v e Af{A{K)), dw{n,y) = ^vcl{ I \\p-p'\\d'n{p,p')\'K e P{ii,v)} 

Jp,p'&A{K) 

where P{^^u) denotes the set of probability distributions on A{K) x A{K) having /i,z/ for 
marginals. 
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Definition 7.5. Let : A{K x I) ^ Af{A{K)) be defined by 

iel : z{i)>0 

with z{i) = J2keK ^(^^ ^) '^"'^ Pi^^ ^)'^ ~ J^^^ ^ such that z{i) > 0. 

^' is a disintegration mapping. Imagine {k, i) is selected according to z and only i is observed, 
then for each i one can compute the conditional p{z,i) on K. ^{z) gives the law of this 
posterior on K. Using the z variable and the desintegration the definition of NR(p) can 
now be read as follows: 

X G NR{p) ^ V/c G K,Yz(k,i) =/ and ^(z) G Af{B-^{p*)). 
Let us now recall a few properties of ^ . 

Definition 7.6. Af{A{K)) is endowed with the convex order denoted < and defined by 

IJL < V <;=^ j (/)dfj, < J 4>dv for all l.s.c. convex (p : A{K) JRU {+oo}. 

For instance, given p and p' in A{K) the Dirac measure S{p+p')/2 is smaller than the 
average 1/2 6p + 1/2 6pi. 

Lemma 7.7. ^ is continuous and convex. 

Proof. See e.g. [21] Lemmas 4.16 and 4.17. □ 

The convexity of ^ with respect to the convex order can be interpreted as the positive 
value of having more information. Before going back to the main proof, let us also recall the 
classical relationship between convex order and martingales. 

Theorem 7.8 (Blackwell,1953 [2]). For all G Af{A{K)), if U and V denote respectively 
the supports of and v, then 

{ 3F -.U ^ A{V) s.t. : 
li<u -i^ I 'iueU, Ylvev vF{u){v) = u, 

The condition on the right hand side can also be seen as follows: /i is the law of some random 
variable Xi with values in A(i^), v is the law of some random variable X2 with values in 
A{K), and we have the martingale condition: IE{X2\Xi) = Xi. 

We will make a change of variable in the expression of <I>q,(/). The definition of NR{p) 
can now be stated as: 

x G NR{p) ^6p< "^{z) and ^(z) G Af{B-^{p*)). 
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Definition 7.9. We define the set-valued maps: 

H: A{K) ^ A/(A(K)) 

peA{K) ^ Hip) = {t,eAf{B-^{pB))\J^^j,^p'di,ip')=p}, 

R: Af{A{K)) z4 A{KxI) 

H e Af{A{K)) R{fi) = {ze A{K x I) ■ < 

We define similarly H : A(L) z4 Aj(A(L)) and R : Aj(A(L) A{L x J) (keeping the same 
letters H and R for simplicity). 

H{p) is the set of probabilities ^ over B-^{pB) with mean p, equivalently it is the set of 
probabilities fj, over B~^{pB) such that Sp < fi. Hence H{p) is the set of splittings at p such 
that all posterior keep the same weight on each recurrence class. The set R{n) can be seen 
the set of "strategies" z with disintegration no more informative than /x. 



Lemma 7.10. For fj, G H{p) and z G we have ^'(z) G H{p). 

Proof. Write p* = pB. Consider the map ^ equal to on B''^{p*) and to +oo elsewhere. It 
is convex and l.s.c. since B'~^{p*) is closed and convex, and 

0< I ^d^{z) < j ^d^ = 0, 

which implies ^{z) G Af{B~^{p*)). Moreover, for all k E K, the map p ^ p^ being also 
convex and l.s.c, 

j p^d^{z){p) < J p^dix{p) =p^, 
which implies / pd^{z)(j)) = p since it belongs to A{K). □ 



It follows from the above definitions and the characterization of NR(p) that for all a G 
[0, 1] and /, one can rewrite as follows: 

^aif)ip,q) 

= sup inf ( aG{z, w) + {1 - a) / f{pM, qN)d^{z) (g) '^{w){p, q) 

z s.t.-^{z)eH(p)w s.t.^{w)eH{q) \ J A(K)xA{L) 

= sup sup inf inf aG{z, w) + {1 - a) / f{pM, qN)d^{z) ® ■^{w){p, q) 

IJieH(p) zeRiij) veH{q) weR{v) \ Ja{K)xA{L) J 

where G{z,w) = ^^k,i,i,j)eKxLxixJ^i'^''i')^i^^j)9ik,l,i,j)- 

We will simplify the above expression as follows. 

Proposition 7.11. Define i/) = sup^^^^^^ inf^^^^j^^ ^(z, u)). Then V is 1-Lipschitz for 
dw and is I-concave and Il-convex on Af{A{K)) x Aj(A(L)). Moreover: 

^aif){p, q) = sup inf aV{n, z/) + (1 - a) / /(pM, qN)dn ® i^{p, q) 

lM&H{p)u&H{q) \ JA{K)xA{L) 
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Proof of Proposition 7.11: Using that / is //-convex on B ^{p*) x C ^{q*) and that 
C^^{q*) is invariant by N, we have for all w G /i(i^) 



f{pM,qN)d-^{z)®^{w){p,q) < I f{pM,qN)d-^{z)®v{p,q), 
A{KxA{L) Ja{K)xA{L) 

which implies the following equality 

f{pM, qN)d'^{z) '^iw){p, q) ] 



inf inf \ aG{z,w) + {1 — a) / 

inf inf aG(z, w) + {\ — a) i 



A{K)xA(L) 

fipM, qN)d^{z) (g) v{p, q) 



u£H{q)w£R{u) \ JA{K)xA(L) I 

since one can always choose v = ^(w) in the right-hand side. A similar equality holds for 
^{z) and n, and this implies that for the moment we have proved: 
For all a G [0, 1], / continuous and {p,q) £ ^{K), 



^a{f){p-,q) = sup sup inf inf I aG{z,w) + {1 — a) / 

f^eHip) 2Gi?(At) ueH{q) weR(u) \ J 



fipM, qN)diJL®v{p,q) 
A{K)xA{L) , 



Let us now study the properties of the set- valued map R. 

Claim. The set-valued map /? : /i — t- /?(^) is non expansive from {Af{A{K)),dw) to 
{A{K X /), ||.||). Moreover, for ah /3 G [0,1], and fi, fi' G Af{A{K)), 

mif^) + (1 - mif^') c R{(3fi + (1 - 

Similar properties also hold for R : u ^ from Aj(A(L)) to A(L x J). 

Proof of the claim: We first prove the non expansive property. Fix ^ and ^' G Aj(A(/f)), 
and z G R{fJ.), we have to find z' in R{fj.') such that \\z — z'\\ < dwifJ-, fJ-')- The idea is to link 
^{z) to fi via Theorem 7.8, then to link and /x' via Kantorovich duality formula, and finally 
to define z' using both links. 

Since ^{z) < fi, there exists a map F : A{K) Af{A{K)) such that for all x G A{K), 
F{x) is centered in x and ^ = ^^^i z{i)F{p{z,i)). Let U,V denote respectively the supports 
of ^ and /x', and consider by Kantorovich duality a probability vr on ?7 x y with marginals 
such that Y.{u,v)(iUxV 11^ ~ v\\'k{u,v) = dwilJ-, fJ-')- For u £ U, let 7r{u) = {^^^^)vev 
denote the conditional law on V given u induced by vr. Because the second marginal of vr is 
n', we have Ylueu fJ-{u)TT{u) = fj.' . We define, for all i in /: 

p'{i) = ^ f m'{i){v) G A{K) with m\i) = ^ F{p{z,i)){u) Tr{u) G A{V), 
vev ueU 

and z' G A{K x /) by z'{k,i) = z{i) p'{if V/c G /C,Vi G /. 
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We have X^jg/ z{i)m'{i) = fi', and for each i the probability m'{i) is centered in p'{i). By 
construction ^'(z') = Yli^i ^i^)^p'(i)- ^^^^ imphes by Theorem 7.8 that ^{z') < /j,'. Using 
Jensen's inequahty, we have 

Ik - z'W = ^2;(i)||p(z,i) -p'{i)\\ 

= E^WIlE^^(^(^'*))(^)- E vF{p{z,i))(u)niu){v)\\ 
iel u&U {u,v)£UxV 

= E^WII Y.^u-Y,vir{u){v))F{p{z,i)){u)\\ 

v]\\u — v\ 



<5]z(i)5]F(p(z,i))(n)5]7f(n)( 
= E E(E^(Wp(^,^))(n))vf(n)(^; 

= E E - v\\ 



ueUvev 

{u,v]\\u — v\ 



= E 

(n,ij)G!7xy 

= dw{fJ',p') 

For the second assertion, let z G Rip) and z' £ then since ^ is convex, 

^(/3z + (1 - I3)z') < ^^(z) + (1 - </3fi + {l- f3)p'. 

This ends the proof of the claim, and we can conclude the proof of Proposition 7.11. 
The law /i being fixed, let us now consider the map 



[z, i^) G R{p) X H{q) inf ( aG{z, w) + {1 - a) [ f{pM, 

weR{u) \ Ja{K)xA{L) 



qN)dp. (gi iy{p, q) 



As an infimum of linear continuous functions, it is concave and upper semi-continuous with 
respect to z on the compact convex set R{p)- On the other hand, the map 



f f{pM,qN)dfi^u{p,q) 
Ja(K)xA(L) 



V 

lA{K)xA{L) 

is linear and using the second point in the above claim, the map 

v — inf G{z, w) 

is convex with respect to u. Indeed, let /3 G [0,1] and z^, z^' G H[q). By compactness, there 
exist w*,w'* which minimize G{z, .) on i2(i^) and R{i'') respectively. Since Pw + (1 — I3)w' G 
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p mill G(z, w) + {l- /3) mill G{z, w) = fiG{z, w*) + {l- l3)G{z, w'*) 

= G{z,l3w* + {l-/3)w'*) 
> min G(z,w). 

It allows to apply Sioii's Minmax Theorem (see [24]) to conclude that 

^a{f){p,Q) = sup inf sup inf \ aG{z,w) + {1 — a) / f{pM,qN)d^(S)i>{p,q) 
fieH{p) ueH(q) zeR(p) weR(u) \ Ja{k)xA{l) ^ 

Using the claim about R and that z — )• G{z, w) is 1-Lipschitz, the function V is 1-Lipschitz for 
dw- Moreover, using the same proof as above, V is /-concave and //-convex on Af{A{K)) x 
Aj(A(L)). The preceding expression becomes 

'^M)iP,Q)= sup inf iaV{n,i^) + {l-a) f{pM,qN)d^i®u{p,q)\, 

IJ.&H(p)u&H(q) \ JA{K)xA{L) J 

and this concludes the proof of Proposition 7.11. □ 



It remains now to study the properties of the set-valued map H. Let us start with the 
following lemma which proves a kind of "triangle equality" for the map S. 

Lemma 7.12. For any pair {p*,p'*) in P* , there exists an affine map L = Lp*^pi* from 
B-^{p*) to B-^{p'*) such that L{p*) = p'* , and for all {p,p') G B-^{p*) x B-^{p'*) ' 

S{p,L{p)) = \\L{p) -p\\ = \\p* -p'*\\, 

S{p,L{p)) + S{L{p),p') = S{p,p'). 

Note that using matrices of example F section 4, the map L is just a homothetic trans- 
formation as illustrated below. 



c 
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Proof. Note at first that B'^ip*) = {p £ A{K)\X{p) = A(p*)}. Since p*,p'* £ P*, we have 
for all r = 1, ..,rM, P*^. = p|* = p*{r). Define the map L by 



rM 



L{p) = Y,Kv'*rv\. 



r=l 



It is easily seen that it defines an affine map with values in B ^(p'*). Moreover 

\\L{p)-p\\= E E 1(^-1)1^'+ E w*rY.p*( 

r:\(p'Y>Ok&K(r) ^' ' r:A(p)'-=0 k&K(r) 

= E w*y-kp*y\+ e ^^p'*y 

r:X(p*Y>0 r:A{p*)'=0 

= \x{p'*Y - x{p*Y\ = \\P* -p'*\\ = s{p*,p'*)- 

r=l 

For the last equality, note that by construction 

S{p,Lip)) = \\X{p')-X{p)l 

SiL{p),p)= Hp)\\P\r - p'lrW, 

r:\{p'YX{pY>0 

SO that 

S{p,L{p)) + S{L{p),p') = S{p,p'). 

□ 

Using the last lemma, we now prove that in some sense H is "non expansive with respect 
to S\ 

Lemma 7.13. For all p and p' in A{K) and for all fi = Yln=i '^n^Pn ^ B{p), there exists 
V = "^^^i OinSp'^ G H{p') such that 

N 



Y'^nSiPn,p'n) = S{p,p) 



n=l 

In particular p ^ H{pB) is 1 - Lip schitz from (A(i^), ||.||) to (A(A(i^)), dpi/). 

Proof. Let /i = X]^=i '^n<Jp„ £ H{p) and p' E A{K). Let us denote p* = pB and p'* = p'B. 
Note that p\y. = Yl,n=i'^nPn\r since for all n = 1,..,N, A(p„) = X(j)). Let L = Lp*^pi* be 
the affine map given by Lemma 7.12. Define pn = L(pn) for n = 1,..,N, so that L{p) = 
X]n=i N^nPn- From Lemma 7.12, for all p" £ B~^{p'*), we have 

S{Pn,p") = S{pn,Pn) + S{pn,p")- 

For r = 1, ..,rM, using Lemma 9.1 in the simplex A{K{r)) (seen as a subset of A{K)), there 
exists (6r,n)n=i,..Af G ^{Kir)) such that J2n=i,..,N ^ribr,n = p[^ and 

E "^IIPn|r - ^n,r-|| = II E "n^njr " pjr II ■ 
n=l,..,Ar n=l,..,Af 
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If \{p'Y > 0, then 

^ ^ (^nPn\r ~ ^ ^ OinPn\r ~ P\r- 
n=l,..,N n=l,..,N 

Let p'^ = Er=i,..,rM Kp'Ybr,n, then X;^=i "nP^ = p' and 

^a„5'(p„,p;) = Xip'y\\Pn\r-Pn\r\\ 

71=1 n=l r:A(p')''>0 

N 

n=l r:X{p'Y>0 

= Yl KPr\\P\r-p\r\\=SiLip),p'). 
r:\{p'Y>0 

It follows that 

N N 

'Y'^nS{pn,Pn) = ^ «„ (^(pn , p„) + S{pn,Pn)) 
n=l n=l 

= S{jp,L{p)) + S{L{p),p') = S{p,p'). 



□ 



We now conclude the general proof of Proposition 5.13. Let e > 0, p' G ^{K) and 
II* G H{p) such that 



inf [aV{ii\v) + {l-a) [ 



fipM,qN)dfi*^iyip,q) > ^>,(/)(p, g) - e 
A{K)xA{L) ) 

If /i* = Oni^pn, using Lemma 7.13, there exists /x' = '^"'^pJi ^ H{p') such that 



^an5'(pn,Pn) = S{p,p'). 

n=l 

Let z^* G //(g) such that 

ay(/i',i/*) + (l-a) [ f{pMAN)dii' ®u*{p,q) 

Ja{K)xA{L) 

< inf {aV{fi',u) + {l-a) [ f{pM,qN)dfi'(l?)u{p,q)]+e<<^a{f){p,q)+e. 

ueHiq) \ Ja(K)xA{L) j 

We have for ah p,p' , f{pM,q) - f{p'M,q) < S{pM,p'M) < S{p,p'), and dw {f^* , fJ-') < 
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Hn^nWPn - p'nW < Y.N '^nS {pn, Pn) = S{p,p'). Consequently, we have 

'^a{f)ip,q)-'^a{f){p',q) 

<a{V{fi*,u*)-V{fi',iy*)) + {l-a) [ f{pM,qN)d{fi*-f,')0u*{p,q) + 2e 

J/\{K)xA{L) 

N 

< adw{fi*,fi') + (1 - a) / y^an{f{PnM,qN) - f{p'^M,qN))du*{q) + 2e 

N 

< aS{p,p') + (1 - a)Y,anS{pn,Pn) + 2e = S{p,p') + 2e. 

n=l 

We conclude that ^a{f){p, Q) — ^a{f){p', q) < S{p,p'). The symmetric property follows by the 
same method. In order to conclude the proof, it remains to prove that is /-concave and 

//-convex on B^^(p*) x C^^{q*), but this follows directly from the fact that for all /3 G [0, 1] 
and p,p' £ A{K), 

(3H{p) + (1 - /3)//(y) C H{Pp + (1 - P)p'), 
and the similar property for q, q' G A(L). 

8 Open Questions 

Note that the Lipschitz constant of 3 for the non revealing values is essentially used to obtain 
the uniform convergence of the non revealing value functions vt- We do not know how to 
prove this uniform convergence in a simpler way, neither if one can obtain a better Lipschitz 
constant for the non revealing values. 

The Maxmin of the infinitely repeated game with incomplete information on both sides was 
proved in [1] to be equal to cav/vex//n. In the present model, it may be asked if the Maxmin 
is equal to cav/vex//?;. One may hope to prove that Player 2 can defend this quantity by 
combining the methods developed in the present work and the proof in [14]. It should be 
more complex to determine what can be guaranteed by Player 1. Even if one proves that 
Player 1 can guarantee the limit v{p, q) in the non revealing game (and consequently that 
the non revealing game has a uniform value), this does not imply a priori that Player 1 
can guarantee vexuv in the original game. An other idea would be to consider the "semi- 
revealing" game where only Player 1 is restricted to play a non revealing strategy, then to 
prove that this semi-revealing game has a limit value v'{p, q) that can be guaranteed by Player 
1, but then there is still little hope to be able to link cav/u' with vexjjv. 

The results of Aumann and Maschler, and of Mertens and Zamir were extended to the case 
of imperfect observation of the actions. In the same way, our model can be extended to the 
case where instead of observing the past actions played, each player observes a stochastic 
signal depending only on the past actions (state independent signalling). The notion of non- 
revealing strategies still makes sense, so conceptually one may hope to use the same ideas to 
prove the convergence of the value functions in this case. Note that a difficulty in this model 
is that even if we fix the strategy of a player, say Player 1, this player is not able to compute 
along the play the belief of Player 2 about the current state in K. And when introducing 
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signals, it is often the case that one has to construct strategies by blocks sufficiently long to 
allow for statistical tests. How these blocks would interfer with the blocks where non revealing 
strategies are used as in point (5) of the proof of Theorem 6.1 is unclear, and this seems a 
highly technical problem. 

An important extension of repeated games with incomplete information on both sides is the 
case of general, possibly correlated, probabilities for the initial pair of states {k,l). The 
results of Mertens and Zamir [14] were actually stated within this more general framework. 
This suggests the following generalization of our model: we are given an initial probability 
vr G A(K X L) and a Markov chain Zf = {kt, It) with values in x L. At each stage. Player 
1 observes kt and Player 2 observes It- It is still possible to define the appropriate notion 
of convex and concave hull as done by Mertens and Zamir. However, the definition a non- 
revealing strategy should be adapted. Indeed, even if Player 1 does not use his information 
on the state, the beliefs of Player 2 on the recurrence classes of the chain zt may evolve. This 
is left for future research. 



9 Appendix 

Lemma 9.1. (Laraki [11]) K being a finite set, consider the simplex A{K) = {{p'^)k&K ^ 
' 'l2keKP^ ~ ^} endowed with the L^-norm \\p — p'\\ = X^fceK Ip'^ — . 
Consider a convex combination p = "^g^g ^sPs ^(-^); with S finite, > and 

^^g^-As = 1. Then for every p' in A{K) there exists {p's)ses ii^ A{K)^ such that: 

p' = J2^^p's ^^'^ X]-^*"^* = 

SG5 seS 

Proo^: For each k in K, denote by Ck the unit vector (e^')fc' in A{K) such that = 1. Define 
= {k e K,p'^ > p''} and K~ = {k e K,p"' < p''}. Define also: A =def Y^keK+^P'^ ~P^) = 
YkeK-ip'' -p'^) = \\\p-p'\\^ and introduce 6 G A(K+) by 9^ = {p'^ -p^)/A for all k in . 
One can now define for all s: 

P's=Ps+Y. 4(p'-p")(^-^k). 

keK- ^ 

1) For all k in K~, we have pf = p\pl^ jp^ > 0, and p'^ G A{K). 

2) Es ^sP's =P+ T.keK- (/ -p"'){0-ek)=p + Ae + ^^ei^- + P'^^k =p + p'-p = p'. 

Y.keK- ^{P^ - P'^)) - T^keK- ^{P^ -p'^Yk- We have a difference of vectors 

with disjoint support, so the norms add up and: \\ps — p'sW = "^Ylik&K- ^(P^ — p'^). This is 
true for each s, and we obtain: 

Y,UPs-p'A = 2 Y.{p''-p'^) = \\p-p'\\. □ 

s€S k&K- 

^The proof can be deduced from Lemma 8.2 in [11], which deals with general measurable spaces. 
The simpler but less general proof given here is easily deduced from the version written in R. Laraki's 
PhD thesis. 
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